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(54) A spread spectrum watermark for embedded signalling 



(57) A watermark is embedded into audioAndeo/- 
image/multimedia data using spread spectrum method- 
ology. The watermark is extracted from watermarked 
data without the use of an original or unwatermarked 



version of the data by using spatial or temporaJ local 
averaging of the frequency coefficierTts of the water- 
marked data. 
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Description 

Field of the Invention 

The present invention relates to digital watermark- 
ing of data including audio, video, image and multimedia 
data. Specifically, the invention relates to the extraction 
of a waternnark of embedded data from watermarked 
data without using an original or unwatermarked version 
of the data. 

Background of the Invention 

The proliferation of digitized media such as video, 
image, video and multimedia is creating a need for a 
security system which facilttates the identification of the 
source of the material. 

Context providers, i.e. owners of works in digital 
data form, have a need to embed signals into 
audio/video/image/muiti media data which can subse- 
quently be recorded or detected by software and/or 
hardware devices for purposes of authenticating copy- 
right ownership, control and management. 

For example, a coded signal might be inserted in 
data to indicate that the data should not be' copied. The 
embedded signal should presen/e the image fidelity, be 
robust to common signal transformations and resistant 
to tampering. In addition, consideration must be given to 
the data rate that can be provided by the system, 
though current requirements are relatively low - a few 
bits per frame. 

In U.S. Patent Application 08/534.894. filed Sep- 
tember 28, 1995. entitled "Secure Spread Sp>ectrum 
Watermarking for Multimedia Data" and assigned to the 
same assignee as the present invention, there was pro- 
posed a spread spectrum watermarking method which 
embedded a watermark signal into perceptually signifi- 
cant regions of an image for the purposes of identrfying 
the content owner and/or possessor. A strength of this 
approach is that the watermark is very difficult to 
remove. In fact this method only allows the watermark 
to be read if the original image or data is availattle for 
comparison. This is because the original spectrum of 
the watermark is shaped to that of the image through a 
non-linear multiplicative procedure and this spectral 
shaping must be removed prior to detection by nnatched 
filtering and the watermark is inserted into the N largest 
spectral coefficients, the ranking of which is not pre- 
served after watermarking. Thus, this method does not 
allow software and hardware devices to directly read 
embedded signals. 

In an article by Cox et aL, entitled " Secured Spec- 
trum Watermarking for Multimedia" spread spectrum 
watermarking is described which embed a F>seudo-ran- 
dom noise sequence into the digital data for watermark- 
ing purposes. 

The prior art watermark extraction methodology 
requires the original image spectrum be subtracted 



from the waternnark image spectrum. This restricts the 
use of the method when there is no original image or 
original image spectrum availatjle. One application 
where this presents a significant difficulty is for third 
5 party device providers desiring to read embedded infor- 
mation for operation or denying operation of such a 
device. 

The present invention extends the earlier work of 
Cox et al to allow the reading or extraction of embedded 

10 signals by devices which do not contain original data, 
e.g. original images. 

In U.S Patent No. 5.319.735 by R.D. Preuss et al 
entitled "Embedded Signalling' digital information Is 
encoded to produce a sequence of code symbols. The 

IS sequence of code symbols is embedded in an audio sig- 
nal by generating a corresponding code signal repre- 
senting the sequence of code synrt>ols. The frequency 
components of the code signal being essentialiy con- 
fined to a preselected signalling band lying within the 

20 bandwidth of the audio signal arwJ successive segments 
of the code signal corresponds to successive code sym- 
bols in the sequence. The audio signal is continuously 
frequency analyzed over a frequency band encompass- 
ing the signalling t>and and the code signal is dynanni- 

25 dally filtered as a function of the analysis to provide a 
modified code signal with frequency component levels 
which are, at each time instant, essentially a prese- 
lected proportion of the levels of the audio signal fre- 
quency components in corresponding frequency 

30 ranges. The modified code signal and the audio signal 
are combined to provide a composite audio signal In 
which the digital information is embedded. This compo- 
nent audio signal is then recorded on a recording 
medium or is otherwise subjected to a transmission 

35 channel. 

Summary of the Invention 

The present invention overcomes the limitations of 
40 the prior systems by using spread spectrum technology 
to embed watermark data or information into predeter- 
mined locations in an image. 

More specifically, the invention provides a system 
for extracting a watermark from watermarked data with- 
45 out using an original or unwatermarked version of the 
data. 

The preferred method of watermarking extraction is 
to use a spatial or temporal local average of the fre- 
quency coefficients of the watermarked data to deter- 
so mine the watermark. The frequency coefficients of a 
two-dimensional neightxjrhood in two-dimensional 
watermarked data (e.g. an image) ,^ for example, are 
analyzed to reproduce the entire watermark. This Is 
possible since the watermark is embedded into the data 
55 using spread spectrum technology which places the 
watermark throughout the data. 

The invention is applicable to the watermarking of 
audio/video/image/multimedia data. 



2 



3 



EP 0 828 372 A2 



4 



The invention will be best understood when the fol- 
Icjwing description is read in conjunction with the 
accompanying drawing. 

R rief Description of the Drawings 

Figure 1 is a schematic block diagram of a method 
of inserting a watermark into an image; 

Rgure 2 is a graphical representation of the image 
spectrum and shaped watermark spectrum; 

Figure 3 is a schematic block diagram of a com- 
biner; 

ngure 4 is a schematic block diagram of a method 
of extracting a wateroiark from a watermarked 
image; 

Rgure 5 is a schematic block diagram of a separa- 
tor; 

Figure 6 is a schematic block diagram of a spread 
spectrum system for use In watermark insertion; 

Figure 7 is a schematic block diagram of a spread 
spectrum receiver; 

Figure 8 is an original image to be watermarked; 

Figure 9 is the image in Rgure 8 after being water- 
marked; and 

Rgure 10 shows a 4x4 array indicating the 
sequence of coefficients used to form a one-dimen- 
sional vector. 

Detailed Description 

Referring now to the figures and to Rgure 1 in par- 
ticular, there is shown a schematic block diagram of a 
method for inserting a waterrnark into a digital data, for 
instance an image. In the following description refer- 
ence may be made to image data or images. While the 
invention has applicability to image data and images, it 
will be understood that the teachings herein and the 
invention itself are equally applicable to audio, video, 
image and multimedia data and the term image and 
image data will be understood to include these terms 
where applicable. As used here in watermark will be 
understood to include embedded data, symbols, 
images, instructions or any other identifying information. 

The image 1 0 is first transformed into a spatial fre- 
quency representation 12. for instance by a discrete 
cosine transform (DCT), other transforms such as a fast 
Fourier transform could also be used. The spectrum is 
then analyzed to determine the perceptually most sig- 
nificant components 1 4 and the watermark to be imbed- 



ded 16 is then combined 18 with the perceptionally most 
significant components. The watermark is a pseudo 
random number sequence (PNS) preferably chosen 
from a Gaussian distribution. After being combined, the 

5 modified image is then inverse trarrsformed back into 
the spatial domain to create the watermarked image 20. 

There are different ways to combine the watermark 
with the image spectrum. In the preferred embodiment, 
the watermark components. W^, are added to the fre- 

w quency coefficients. //, in a non-linear manner as 

//=/,+a/,lV, (1) 

where a is a constant typically in the range of 0.1 to 
15 0.01 . In principle, a might also vary as a function of fre- 
quency and perceptijal modeling. Equation (1) can be 
considered a form of spectral shaping. That is . the orig- 
inal Gaussian white spectrum of the watermark Is 
shaped to match that of the image by the second term 
20 in Equation (1 ) prior to addition of the two spectra. The 
constant a serves as a gain control to adjust the relative 
strength of the two spectra. This is graphically shown in 
Figure 2. 

The two stages of the combiner are shown in Rgure 
25 3. The watermark to be embedded into the data is pro- 
vqded as a first input to a spectral shaper 30. The spec- 
ti-um of the image to be watermarked is provided as a 
second input to the shaper 30. The output of shaper 30 
is provided as a final input to summer 32. The spectrum 
30 of the image is provided as the second input to summer 
32. The output of summer 32 is a watermarked spec- 
trum. 

To extract the watermarK the inverse process must 
be applied as shown in Rgure 4. The separator stage 

35 inverts the combiner stage. In order to extract tiie water- 
mark components, Wj, from a possibly distorted water- 
mark image, first subtract the original image before 
dividing by the innage spectral coefficierrts. The latter 
process serves to normalize or equalize the watermark 

40 spectrum back to its original shape. Jhat is 

l^,=(//'-/,ya// (2) 

Specifically, tine watermarked image 40 is ti^ns- 
45 formed by a discrete cosine b-ansform or other transfor- 
mation such at FFT. into a watermarked image 
spectrum 42. The stored original image 44 is trans- 
formed into an original image specti-um 46. 

The waternriarked image spectrum 42 and the orig- 
50 inai image spectrum 46 are provided as inputs to sepa- 
rator 48. The separator, as shown in Rgure 5, subtracts 
the original image spectrum from the watermarked 
inr^ge spectrum 54 to obtain a difference image spec- 
trum prior to normalizing the resultant image. The spec- 
55 tral normalization 56 divides ttie difference image 
spectrum by the image specti-al coefficients a// to yield 
an extracted watermark. 

The extracted watermark is statistically compared 
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with the known inserted watermark to calculate a statis- 
tical confidence level. The statistical confidence level 
provides a measure of whether the externa! watermark 
is the actual inserted watermark. 

In the above described method of extracting a 
watermark, it is necessary to have an original un water- 
marked image. This is both an advantage and limitation 
of the method. It is advantageous because it is difficult 
for a non-possessor of the image to remove the water- 
mark. A limitation of the method is that it prevents a third 
party's software or hardware devices from extracting or 
reading the embedded signal information. The use of a 
Gaussian noise distribution is important for extraction 
using an original image. 

The above described prior art system is a special 
case of a more general spread spectrum communica- 
tion system in which the watermark information is con- 
sidered as the signal and the image is considered as the 
noise. Rgure 6 is a schematic block diagram of a spe- 
cial spectrum communication system for use in water- 
mark insertion. 

In Figure 6 a watermark signal is provided as an 
input to an error correction encoder 60. The output of 
encoder 60 is provided to a spread spectrum modulator 
62. The output of modulator 62 is provided to a spectral 
transformation 64. The output of spectral transformation 
64 is provided as one input to a spectral shaper 66. A 
signal to be watermarked is provided to a spectral trans- 
former 68. The output of the transformer 68 is provided 
as a second input to spectral shaper 66 and to a delay 
70. The output of the spectral shaper 66 Is added to the 
output of delay 70 at a summer 72. The second output 
is subject to an inverse transform 74. The result of the 
inverse transform is a watermarked signal. 

In the prior systems, the object was to embed a sin- 
gle PN (pseudo random number) sequence into an 
image. The information associated with the PN 
sequence was assumed to be stored In a database 
together with the original irriage and the spectral loca- 
tion of the embedded watermark. The locations of the 
watermarked components had to recorded because the 
implementation approximated the N perceptionally most 
significant regions of the watermark by the N largest 
coefficients. However, this ranking was not invariant to 
the watermarking process. The N largest coefficients 
nnay be different after inserting the watermark than 
before intersecting the watermark. 

In order to avoid this problem, the current method 
places a watermark in predetermined locations of the 
spectrum, typically the first N coefficients, However, any 
predetermined locations could be used even though 
such locations should belong to the perceptually signifi- 
cant regions of the spectrum if the watermark is to sur- 
vive common signals transformations such as 
compression, scaling, etc. 

More generally, the information to be embedded is 
a sequence of m symbols drawn from an alphabet A 
(e.g. the binary digits or the ASCI! symbols). This data 



is then supplemented with additional symbols for error 
detection and correction. Each symbol is then spread 
spectrum nrodulated. a process that maps each symbol 
into a unique PN sequence known as a chip. The 

5 number of bits per chip is preset - the longer the chip 
length, the higher the detected signal -to-noise ratio will 
be. Ixjt this is at the expense of signaling bandwidth. 

The spectrum of the PN sequence is white, i.e. flat, 
and is therefore shaped to match that of the "noise", i.e. 

10 the image/vkJeo/audio/or multimedia data into which the 
watermark is to be embedded. It is this spectral shaping 
that must be modified from the prior methods so that the 
extraction process no longer requires the original 
image. To do this. Equation (1) Is modified so that each 

IS coefficient of the watermarked spectrum is scaled by 
the local average of the image spectral coefficient rather 
than the coefficient itself, i.e. 

fi'=fi+aavg{ff)W, (3) 

20 

This average may be obtained in several ways. It 
may be a local average over a two dimensional region. 
Alternatively, the two dimensional spectrum may be 
sampled to form a one dimensional vector and a one 

25 dimensional local average may be performed. The latter 
method was used in experimental results below. The 
average may be a simple box or weighted average over 
the neighborhood. 

f=br video data, temporal averaging of the spectral 

30 coeffk^ents over several frames can also be applied. 
However, since several frames are needed for averag- 
ing at the spectral normalization stage of the extractor, 
the protection of individual video frames taken in isola- 
tion may not be possit>le. For this reason, the present 

35 invention treats video as a very large collection of still 
images. In this way, even individual video frames are 
copyprotected. 

Receiving or extracting a spread spectrum signal Is 
shown in Rgure 7. The watermarl<ed Image, video. 

40 audio or multimedia data is first spectrally normalized 
76 to undo any previously performed spectral shaping. 
The nornnalized signal is then analyzed by a tiank of 
correlators 78A-..78Z. each corrector detecting the 
presence, if any, of a particular PN sequence (one for 

45 each symbol in the alphabet). The decision drcurt 80 
typically selects the correlator with the maximum output 
as the most likely current symbol. More sophisticated 
decision procedures are possible. The sequence of 
most likely current symbols is then provided as an input 

so to an error correction stage 82 which corrects for false 
decisions made by the decision circuit. The output of the 
error correction stage is an extracted watermark signal. 
In order to perform the spectral normalization 76, the 
previously performed spectral shaping procedure Is 

55 inverted. In the present case, the original unwater- 
marked signal is no longer available. Thus, an average 
of the frequency coefficients. avg(fj}, as approximated 
by the average of the watermarked signal, i.e. avgiJi) 
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(4) 



This is approximately tnje since the second term of 
equation (3) is small relative to the first i.e. 



(5) 



The normalization stage then divides each coeffi- 
cient (/,") in the received signal by the local average avg 
{fi) in the neighborhood. 

That Is. 



avgifi") avgUi) 
avg{ff) ' 



(6) 



The first term, on the tight hand side (RHS) of 
Equation (6) , 



Equation (6) which is not present when using the prior 
art method. 

In order to determine the closeness of the approxi- 
mation of Equation (4). the equalization process was 

5 repeated using the original shaping coefficients, i.e. 
avg(fi), instead of avg {fj'). The correlator response 
increase from 0,125 to 0.15. suggesting that a loss of 
approximately 20% was incurred due to this approxima- 
tion. Of course, this loss is strongly deperxlent of the 

10 local smoothness in the spectra of an image and may 
vary significantly from image to image- 
In summary, the present invention provides a modi- 
fication to existing digital watermarking methods in 
which the original data was required for watermark 

75 extraction thereby enabling watermarking extraction in 
the absence of an unwater marked or original data. The 
present invention uses local spatial and/or temporal 
local averaging of the frequency coefficients. The result 
is extraction of the watermark with very high confidence. 

20 While there has been described andi illustrated a 
system for inserting a watermark into and extracting a 
watermark from watermarked data without using an 
unwatermarked version of the data, it wilt be apparent to 
those skilled in the art that variations and modifications 

25 are possible without deviating from the broad principles 
and teachings of the present invention. 



is considered a noise term. It was not present in the 
prior systems because access to the unwatermarked 
coefficients allowed this term to be removed. The sec- 
ond term a W/ is the original watermark signal which can 
now be detected using conventional correlation. 

Figure 8 shows an original image before being 
watermarked. Rgure 9 is the same image after being 
watermarked in accordance with the teachings of the 
present invention. 

The watermark in Figure 9 was inserted using a 
gain of a = 0.1 and a chip length of 10,000. The first 
1 0,000 coefficients of the original image were extracted 
in the sequence shown in Figure 10 to form a one 
dimensional vector. A block average was then com- 
puted over a rectangular window of +/- 3 coefficients. 
The same procedure was applied at the extraction 
stage. 

The correlator responds to randomly generated PN 
sequences with one such sequence being set to the 
originally inserted sequence indicated. A' very strong 
and unambiguous response on 0.125 was detected for 
a particular PN sequence. For uncorrelated water- 
marks, tfie correlator output is approximately Normally 
distributed with a variance of 1/(N-2). where N is tiie 
length of the watermark. Thus, for N= 10.000. the stand- 
ard deviation is 0.01 and the correlation response of 
0.125 represents over 12 standard deviations. A 
response of approximately 30 deviations was achieved 
with the prior art method using a watermark length of 
only 1 ,000. The reduction in signal-to-noise ratio is due 
almost entirely to the first term of the right-hand side of 



Claims 

30 1 . A method for inserting a sequence of symbols into 
data to be watermarked comprising the steps of: 

spread spectrum modulating each syrrtbol of 
the sequence of symbols by mapping each 
35 symbol into a PN sequence; and 

embedding each PN sequence in predeter- 
mined coeffidertts in the data. 

2. The method as set forth in claim 1 . further compris- 
40 ing the steps of: ^ 

obtaining a spectrum of each PN sequence; 
and 

shaping the spectrum to match the spectrum of 
45 the noise. 

3. A method as set forth in claim 2. where said shap- 
ing the spectrum is performed by temporal or spa- 
tial local average of frequency coefficients. 



50 



ss 



4. A method of extracting a watermark from water- 
marked data comprising the steps of: receiving 
watermarked data: 

spectrum normalizing the watermarked data to 
generate a normalized signal; 
correlating the normalized signal with predeter- 
mined PN sequences corresponding to prede- 
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termined symbols to provide correlated signals 
for each predetermined PN sequence: 
deciding which con-elated signal is most likely a 
currwt symtx5l; and , 

extracting a sequence of most likely current 
symbols corresponding to the watermark. 



10 



15 



20 



25 



30 



35 



40 



45 



50 



55 



6 



EP 0 828 372 A2 



WATCBMARK 







OCT 



1 0 



IMACT 
SPECTRUM 



1 2 



PKRC3BPTUALI.Y 
SIGNIPICAMT 

RKGIONS 



1 4 



COMBINER 



1 8 



INVKRSZ OCT 




FIG. t 



POWER 




IMAGE SPECTRUM 



SEATED WATERMARK 
S?ECTRUM 



FREQUENCY 



FIG. 2 



EP 0 828 372 A2 



3 0 



3 2 



WATERMARK 



IMAGE 
SPECTRUM 













SPECTRAL 
SHAPING 











•i 


i 





WATEJ^MARKED 
SPECTRUM 



PIC, 3 



WATERMARKED 


IMAGE 






DOT 


\ 





4 0 



WATERMARKED 

IMAGE 

SPfiCTRUM 



4 2 



4 4 



ORIGINAL 
IMAGE 



4 6 



ORIGIHAL 
SPECTRUM 



SEPARATOR 



WATERMARK 



5 0 



EXTRACTED 
WATERMARK 



COMPARATOR 



5 2 



FIG. 4 



STATISTICAL 

CONFIDENCE 

LEVEL 



8 



EP 0 828 372 A2 



5 4 



WATERMARKED 

IMAGE 

SPECTRUM 




5 6 





SPECTRAL 
NORMALIZATION 









EXtRACTED 



ORIGINAL 

IMAGE 

SPECTRUM 



FIG, 5 



WATERMARK 
SIGNAL 



6 2 



6 4 



ERROR 




SPREAD 


CORREC- 




SPECTRUM 


TION 


— 


MODULA- 


ENCODER 




-TOR 




6 0 



AUDIO /VIDEO 

IMAGE 

SIGNAL 



6 8 



SPECTRAL 
TRANSFORMER 



6 6 



^ 


SPECTRA! 
SU&FING 







7 0 



7 4 




INVERSE 


TRANS- 


FORM 




2 


r 



WATERMARKED 
AUDIO /VIDEO 
IMAGE 



FIG. 6 



9 



EP 0 828 372 A2 



7 8 A 



WATERMARKED 
DATA 



7 6 



SPECTRUM 
NORMi^LIZATION 



8 0 



EXTRACTED 
WATERMARK ^ 
SIGNAL 





CORRELATOR 











DECISION 
-IRCUIT 








— ^ 


CORRELATOR 















. 7 8 B 
* 






7 a z 














CORRELATOR 


. ^ 






8 2 






ERROR 
CORRECTOR 













FIG, 7 



1 


4 


5 


1 6 


2 


3 


6 


1 5 


9 


a 


7 


1 4 


1 0 


1 1 


1 2 


1 3 



FIC. 10 



10 




Figure 8 



11 



EP 0 828 372 A2 




Figure 9 



12 



